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ABSTBACT 

This final report summarizes research findings and 
presents a model for generative computer assisted instruction (CAI) 
vith respect to its usefulness in the classroom envirpnment* Methods 
used to individualize instruction, and the evolution of a procedure 
used to select a concept for presentation to a student vith the 
generative CAI system are described. The model served as the basis 
for the design of a CAI system, teaching problem* solving in an 
introductory course in digital systems design. The system 
individualizes the instruction vhich each student receives in 
accordance vith its record of his past performance. In addition, a 
heuristic technique is used to determine the best path for each 
student through the tree of course concepts. The refinement of this 
method of concept selection is described. An evaluation of the 
generative CAI system and results of classroom usage are also 
presented. (Author/DAG) 
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ABSTRACT 



This paper describes a model for generative computer -assisted instruction. 
This model has served as the basis for the design of a CAI system used to teach 
problem-solving in an introductory course in digital systems design. The system 
individualizes the instruction which each student receives in accordance with 
ICS record of his past performance. In addition, a heuristic technique is used 
to determine the best path for each student through the tree of course concepts. 
The refinement of this method of concept selection is described. An evaluation 
of the GCAI system and results of classroom usage are also presented. 
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1. 'INTRODUCTION 

Over the past few years . we have W ejcploring the concept of gemwative 
computer assisted Instruction (GCAI). This paper summarizes the findings of 
this research with respect to the usefulness of a generative CAI system in a 
claefiroom environment. The paper de8crli)e8 rethoda used to individualize 
instruction and the evolution of a procedure used to select a concept for 
presentation to a afJdent working with the CAI ^stam. 

This paper is concerned with problem oriented system**. In a typical drill 
4nd practice system values are generated and substituted ft* variables in a 
prestored question format. The values of the variables are constrained so that 
the resulting problem is meaningful and of an appropriate difficulty level; In 
this way some of the semantic content of a question Is provided. In a problem 
oriented generativ-3 system, a problem is generated from a gremnar or other 
suitable structure. Both the syntax and semantics of a problem are detemined 
by the system. Thus, a richer variety of problems can be generated. In addi- 
tion, the system has the potential of controlling the generation piwess so that 
it tan tailor the difficulty and area of emphasis to suit the individual student. 

A generative CAI system must have the capability of solving the problems 
that it generates. Usually, the problems will cover a specific approach to 
design csr a solution algorithm. They will be more complex and involved than the 
manipulations performed in a drill and practice environment. Indeed, the 
incorporation of problem solving capability and semantic knowledge indicate the 
need of an Artificial Intelligence (AI) approach to generative CAI design. 
However, the problem solvers needed in CAI applications are less difficult to 
design for the following reasons: 1) they are concerned with specific problem 
types in a specific subject area 2) they are generally nwre algorithmic than 
heuristic 3) they are supplied with problem generation Inforaation Including 
parameters necessary for solution and do not have to extract this Infonnatlon 
from the problem representation. Bohrow's STUDEKT (1) and Winograd's SHRDLO 
(2), for example, must first interpret the problem statement and extract essen- 
tial information prior to obtaining the solution. 

Nevertheless, generative CAI systems have benefited and can benefit further 
from research in airtificial intelligence on natural language understanding, 
question - answering, problem solving, and automatic prograraning (3). Specifi- 
cally, AI can be beneficial in the design of more general solution routines so 
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that it will not be necessary to implement a solution routine for relatively small 
problem classes. AI techniques can also contribute t^ the construction of solution 
routines to help achieve iDore powerful generation systems. On the other hand, 
study of generative CAI and generative techniques can make a contribution to • 
artificial intelligence by providing results, for example, on ways of identifjrlng 
and using control (generation) information to design efficient and practical AI 
systems. 

2. MODEL FOR GENERATIVE CAI (6CAI) 
A* Introduction 

Figure 1 is a block diagram of a GCAI system. To individualize instruction, 
this system needs a model of the course and the student - the concept tree and 
student record respectively. After a concept has been selected and the appropriate 
level of difficulty determined, a problem is generated and presented to the 
student. The subsequent interaction and monitoring of the student's solution 
is a function of his performance. Each of the components in Figure 1 will be 
discussed below. 



Concept 
Selector 



Concept 
Tree 



Concept Problem 



Level 



Generator 



Question/ 
Remedied 



Student 
Record 




.Vpcjajt^ng 



data 



Figure 1. System Block Diagram 
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B. Course Structure 

A course can be modeled as a hierarchical tree structure in which each node 
represents a ^.oncopt and corresponds to the following information: 

1) concept number 

2) concept name 

3) a list of doubles; the first element of each pair is the number of a 
prerequisite concept and the second, a flag indicating whether or not a 
prerequisite . may be called as a sub?routine 

H) the plateau for that concept (how far up on the tree it is), which 
implies its relative complexity 

5) the name of the problem generator for that concept 

6) the name of the solution routine 

7) a list of parameters which are passed from the ppobl^n generator to the 
solution routine 

For example, (C7, BINARY MULTIPLICATION, ((C4 0), (C5 1)), 2, PROBl, BINMUL, 
(levC7, levC5, multiplicand, mult vlier)) means that C7 deals with binary multi- 
plication. There are two prerequisites. One of the prerequisites nay be called 
as a subroutine, (C5 - binary addition). C7 is on plateau 2 of the concept tree-. 
The paraii.eters passed from the problem generator PROBl to the solution routine, 
BINMUL. are the student's levels of proficiency in concepts C7 and C5, together 
with^e multiplicand and mjitiplier for the generated problem. 

The level of proficiency in C7 ('•5< liyCT^ 3) determines the difficulty of the 
problem to be generated and the degree of instruction and Tuonitoring to be 
received by the student v/hile working on a binary multiplication problem. The 
parameter levC5 is needed in the event concept C5 is called as a subroutine. 

This representation is general and has been used in a system for digital 
systems design W and a system for high school algehru (S). In a varied form 
it has also been used for a machine language programming QCAI system (6). 

The digital systems design GCAI course contains twenty-one concepts begin- 
ning with basic logical operations and number conversions (Plateaus 1, 2), 
combinational design and timing diagrams (plateau 3), minijnizatlon techniques 
(plateau sequential design (plateaus 5,6) and register transfer operations 
(plateau 7), 

The arcs of the tree structure (concept tree) represent prerequisite or 
subconcept relations; the latter indicated by a flag. More generally, the arcs 
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can represent other relations of interest. In machine language programming (6^ 
concepts are. small sections of a program, called input pi'imitivea, processiog- 

primitives, and output primitives. The relation of interest is concatenation. 

The nodes of the concept tree represent the primitives and the arcs reptresent 

possible concatenations of the primitives to form "more complex problems. 

C. Problem Generator 

Only two problem generator routines are used for the first thirteen concepts 

in the digital systems course. This is due to the fact that there is a great 
deal of similarity between the parameters needed for these mJtines. Several of 
these routines require a variable number of binary, octal, decimal, or hexadecimal' 
character strings aa parameters. 

A viabl>- model for a problem generator is that of a probabilistic grainaar. 
A context fvce probabilistic grammar is a formal grammar whose irewrlte.^uies have 
a single nonterminal symbol on the left (generating symboi). Moreover, each re- 
write rule has a non-zero probability associated with it such tliat the sum of 
the probabilities of the rules with the s,-:.me generating symbol is one. In order 
to obtain problems of varying degrees of complexity, thti probability of each re- 
write rule is made a function of a student's level of proficiency in the concept 
being studied; rewrite rules leading to more dirficult problems become more 
likely as a student's proficiency increases. The probabU 'ties can also be made 

- functions of the estimated difficulty of the pca?t?.?Uy generated problem. For 
example, the depth of nesting in a logical expression is one indicator of 
difficulty. The probability of rewrite rules which tend to produce nested 
subexpressions exceeding a preset threshold shouJ-vi drop to zero. This makes 
the probabilities context sensitive and k(jeps the problems from getting out of 
hand. 

The grammar for the problem generator associated with each concept consists 
of a 7-tuple of the following format: 

G = (S,N,TjR,P,Z,D) 

where S is the starting symbol, N is the set of nonterminal symbols, T the set 
of terminal symbols, H the set of rewrite rules. P is an array of probabilities 
with three columns since it was preferred to distinguish between only three 
categories of problem difficulty. The student's previous level of performance 
in a problem determines which column of probabilities will be used during the 
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generation of a problem. Each row in P corresponde to a Mwrite rule. The 

- probabilities. associated with rewrite rules having the same generating symbol 

sum to one over each column of the array. 

D is a function which may be used to calculate the difficulty of the 
partial output string and Z is a vector of difficulty thresholds as mentioned 
above. For most problem classes, neither D nor Z were deemed necessary. 

Table 1 is an example of a grammar which generates! logical expressions for 
the concepts which teach truth table formation and the analysis of sequential 
■ circuits. ^ 

The first column in the array P is the vector of prbhabilities for beginning 
students, the second column is for intermediate level students, and the third 
column is for advanced students. The harder operators + (n&ad) , + (nor) , and 
e (exclusive-or) become more likely as a stiident's proficiency increases. Simi- 
larly, the number of distinct variables likely to appear in an expression 
increases with student proficiency. Rules ''iiich expand the expression also 

become more likely. The vector Z specifies that the probability of these two rule 
drops to zero when the length of the expression exceeds the student's proficiency 
level by a factor of twenty. 

Table 1: Probabilistic Grammar Example 
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D. Problem Solvers and Solution Monitors 

In order to solve the problems generated, a set of augmented solution 
algorithms is provided, one per concept. These algorithms were augmented with 
additional information. Each distinct subtask in the algorithms is identified 
and provided with: (1) generation threshold, (2) question format, (3) remedial 
format aet, (4) answer comparator, (5) increment and decrement multipliers. 

The generation threshold indicates a proficiency lev<!9!l beyond which a 
student's solution to the subtask will not be monitored. 

The question format is a skeletal pattern into which problem dependent 
variables will be inserted. The formats are used for generating questions 
concerning each subtask. 

The remedial set is utilized when a student's answer does not match the 
system's solution. It usually consists of formats \Aich provide some explanation 
o( the correct solution procedure in the context of the current problem. Each 
format has an associated threshold level. If a student has a proficiency level 
greater than the threshold, the remedial statement Is not printed out. Hence, 
the completeness of the explanation provided by the remedial set decreases as 
student ororiciency increases. 

The answer comparator determines the degree of correctness of the student 
solution. In most cases, the degree of correctness can be determined by matching 
the solution Btring derived by the system with that provided by the student. For 
some algorithms the comparator consists of an analysis routine which determines 
the type of error being made by the student in order to provide more meaningful 
remedial commentary. The analysis routine may make use of portions of the solution 
algorithm to assist in characterizing the student's error. This allows the system's 
procedural knowledge to be used to make remedial commentary more meaningful. 

The increment and decrement multipliers determine the amount of increase Irx 
proficiency level for each correct solution to a subtask and the decrease for each . 
Incorrect solution. In this way, each subtask Is weighted to produce the desired 
degree of chang-e in level. As part of the increment and decrement, a factor based 
on the student ""s prior performance in a concept is calculated just before the algorithm 
Is called. This factor tends to produce increments which are larger than daerementsT 
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lor^iuudenui who are performing well and vice-vorsa for students performing poorly. 
Thm:, >^ond students are penalized less for an occasional error and poorer students 
.ire required to l)ecome more consistently correct in order to advance to the next 
i'igher level ranpe. 

The aup.mented solution algorithm leads a beginning student carefully through 
a solution procedure while providing more freedom and less interaction for advanced 
student:;. The decree of freedom and monitoring is dynamically adjusted as a student's 
Jovoa of performance changes. Table 2 gives examples of the varying degree of 

inr»v,.,^,>t ion possible. Reference U describes problem generation and solution in 

/UTdteT' detail 4 

TABLE 2: Student Interaction in Problem Solution 

lnit.I«a Level Hanpe 0>-l 

lovvA the logical AND of 73U5,2205 in the base 8: 
TaMt;.' corresponding pairs of digits starting at 
the r ipht : 

What is the binary equivalent of 5? 

jOj 

V/luit is the binary equivalent of 6? 
131 

No, The binary equivalent of 6 is 110. 
Wlidt is the logical AND of 101,110? 

JOO 

Wh«it if; the lop.icdl AND of 5, 6? 

[[ (Note: Here student's level becomes >1) 

Wluit in the lo^^.ical AND of U,0?,,. 

Initial Level Hanjp,e 1-2 

Vorm the lopical AND of A2 ,4,1.04 in the base 16: 
Modify A2.U by adding trailing and/or leading zeros. 
0A2,4 

No. Al'.M sijould ]>e changed to A2,U0 

Modiiy l.CM by adding trailing and/or leading zeros. 

Torni the lor.ical AND of A2.U0,01,CU in base 16: 
Taking corresponding pairs of digits starting at 
the ripht: 

Wiiat is the logical AND of 0,U? 
0 

Wriat is the lopjcal AND of U,C?.,. 

I nitial Level Ranpe 2-3 

rorm the J optical AND of C3.U,1.U93 in the base 15: 
V:..it lii the lopicdl AND of C3 • UOO ,01. U93? 
0'^^.i400 

No. 7;ie logical A^iL of Cd.^OO, 01.493 is Ol.UOO 
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i:. Revirw Koile tind Kxaniple Mode 

Two additional and extremely attractive benefits of this generative approach 
to CAI are the capability to operate in a review mode and an example mode. After 
th<* concept has been mastered (proficiency level >3), all subtasks will be solved 
by the cyr,tnm and no solution monitoring will take place. Only summarizing data 
and iiub-concept solutions will be printed out. A student can use this mode to 
quickly obtain solutions to several of the more difficult problems for later self- 
-.uudy. 

Wlion defiling with a new concept, the student may be uncertain as to some details 
of the :;olution procedure. There is occasionally some ambiguity in questions which 
are posed. Also, while variable formats are allowed for the student's response, 
tiu? r;y,stem is usually better able to handle certain formats than others. To alle- 
viate therie difficulties , an example mode may be entered by setting a flag which 
caur.er. rach question posed to the student to be followed by a "write" statement 
rather than a "read" statement. The net effect is for the system to correctly 
an::wer all of its own questions instead of monitoring the student's response. The 
nludent is able to familiarize himself with the content of the concept and see what 
will I >o expected of him. With each question, the proficiency level temporarily 
Incr-ea^sei' no that the student observes a change in solution monitoring similar to 
thtit which would Le obtained during normal interaction with the system. The example 
ri*ocle opt i on is made available prior to a student's initial use of each concept and 
i,s alniost alway5> used. 

3. SOMK t.'liOiri: FUNCTIONS FOR CAI 
A. Ir.troducrion 

The r.yr;ti^nir. concept selector selects a concept for study based on a student's 
pa:;t performances. The concept selector is the interface between the student and 
: tree of course concepts. It determines which concept is 'best' for a student 
as ar;v p;iven time, and it traces an 'optimal' path for each student through the 
rMurvio concept tree. 

It is difficult to measure the 'goodness' of a particular concept choice and 
tiicro would probably be no consenus among educators as to which particular concept 
i:; for a student at a particular point in time. The goals of concept selection^ 

b.ow<'vor, eive necessarily clear. The time spent by each student in mastering the 
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!*a: ...uiiio !:(.ould bo kepi to a minimum. Thus, he should not be required to waste 
t ori a c:onc*-|.r. already mastered; nor should he be expected to solve a problem 
it l.r hd£i not shown mastery of its prerequisite concepts. The intent is to pace 
iiii!! throup.h the course concepts as quickly as possible without confusing hin. 

There have i^een many studies on the effect of learner versus system control in 
.i (A! environment. These studies seem somewhat inconclusive to date. Rather than 

met the r.tudent to one mode of control, the approach taken was to attempt to 
' t!,« best ioi) of concept selection possible, but to always provide the student 
r^r opportunity to veto any individual system selection or to take control and 
l^ypass system selection entirely. 

The task of concept selection is done in two phases: the first phase deter- 
u.luQi^ the current plateau of the concept tree; the second selects the actual concept 
tc» to worked. 

B. Determining the Current Plateau 

A record of past performance is maintained for each student. This student 
model is shown in Table 3. The twelve items in the table are saved for each of 
the twenty-one concepts of the digital ^.:^;tems design coixrse. As mentioned above, 
the difficulty of the problem generated, and the amount of instruction and monitoring 
a student receives is proportional to his level of competence in that concept. A 
student's level of competence is a real number between .5 and 3. It is incremented 
by a small amount (|ALj< .28) for each correct answer and decremented for each in- 
correct answer. As a student will normally answer more questions correctly than in- 
correctly, his level should gradually increase. 

In addition, for each student, the system saves the current plateau and 4 
performance regulator, called the master average. The master average varies 
between 1.4 and 2.5. A significant increase or decrease in concept level 
(jAL|>.5) results in a proportional change in the master average in the oppo- 
site direction. Consistently good performance will result in a low master . 
average. The master average is rtpdated after each problem is completed. 

t 
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tabu: 3 : Student Record 

) prr»f;tuu, Jovo] for each concopt (T.) 

A wfifhted fwerap,(» of level changes for each concept (WTAVG) 

3 La:;l level change (AL) 

'* The? date each concept was last worked 

5 A sequence number indicating the ordering of system selection 

6 ir of times concepts worked in 0-1 range 

7 Ir of times concepts worked in 1-2 range. 

8 tl of times concepts worked in 2-3 range 

0 Date number of last time each concept worked 

10 # of times student reiected each concept 

11 -fi of times each concept was selected by system and accepted by student 



When a student's average level in all the concepts at his current plateau 
t^xceeds his master average, he is eligible to proceed to the next plateau. 
<'oni3equently , the better a student performs, the faster he vill progress up through 
tiie concept tree. It is possible that a concept will only be worked once or even 
not at all. For this reason, a post-test is given whi(ih consists of a challenging 
question for each concept. As a result of the post-test, the system advances him 
to the next plateau or may require him to review each connept whose corresponding 
poGt-test question was answered incorrectly. The decision whether or not to review 
a concept is based on the number of times he has worked that concept, his level in 
that concept, and his master average. 

If a student is advanced to the next plateau, a pre-test is given to initialize 
the levels for the concepts on that plateau. The pre-test consists of a set of 
three or four questions for each concept. The questions get progressively more 
difficult. A student is given two tries at each question and branched to the next 
^er of questions if both answers are incorrect. The starting level for each con- 
cept is a function of how many of the set of questions dealing with that concept 
wore answered correctly. The starting level ranges from .5 to 1.0. 

The pre-test for the first plateau is also used to initialize the master average. 
Tact more questions answered correctly in the first pre-test, the lower will be the 
starting ir^aster average. 

r. Concept Selection 

There are three or four concepts on most plateaus of the concept tree. A 
preliminary check is first made to ensure that the average level of the prerequisites 
for each concept exceeds the student's master average. If this is not the case, 
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ilw:; concopL Il> i-eplaced as a candidate by its prerequisite with the lowest levels 
Vhlt;. provide:; an additional opportunity for review in case the student has been 
pro/:re£jG.in^ too rapidly. 

iitatlGtic3 are available from each student's record (refer to Table 3) con- 
cerning the Luir date each concept was used, its current level, the last change in 
ievol, and the number of times the concept has been selected by the system or 
r^rnected by tlie student in the past. From this data, many parameter values can be 
^Mr.ulatod for each concept and input to the concept selector. The concept selector 
^iitf-mpts to predict which concept will advance the student the most in the course. 

A scoring polynomial as used by Samuel (7) in his well-known Checkers program 
war; the ba.sis for the concept selector. A scoring polynomial consists of a set of 
parameter valuer-; to be calculated for each considered move arid associated co- 
'•n Icients or wei^rhts. A preliminary training period was undergone in order to 
afMermine a pood set of coefficient values. During actual play, the move which 
i^cored the hip.hest was always made. In a similar manner, the concept which scores 
tho highest among those being considered should be selected for presentation to 
the student. 

Samuel model assumes that there is a single "best move" at any point in the 
parne. Howc^ver, the "best concept" for a given student may depend on his particular 
learner pref rrir-nces and personality traits. Consequently, it was decided to provide 
Tor four separate sets of polynomials and use a student ' s past performance to deter- 
mine which set appeared best suited to him. 

Two trainin^^ periods were undergone by the system in order to define four 
I'corinp, polynomials. Initially, a single scoring polynomial which appeared to 
work reasonably well was derived. This poljmomial is shown in Table 

TABLE 4: Original Independent Variables 

Var. 1 (2*ALx)^*wl wl«l if Lj<0; 

-0 if Lj>p 

Var. 2 (Q-Qi)/(Q-Qm)*w2 w2=1.2 if Q^>0; 

«3 if Qj-O (Concept never worked) 

Var. 3 R*w3 R=«# of uses of a concept at present level range 

(0-1, 1-2, 2-3) /total number of times worked; w3«-l 

Var . 4 I ALx I /Li*w4 w4«l 

. 18 
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Table 4: Continued 
Var. 5 (Tj)+Tp)*w5 w5=.l 

Var. 6 (3-Li)*w6 w6»l if Lj<3 

—25 if Lj>3 

Level of concept I; 

Lj: Concept level; AL; Last level change; for concept I 

I:: Current concept being considered; C^: Minimijm sequence number of 

all concepts on present plateau 
Q:: Current sequence number; Qj-: Sequence number of last time concept 

I was worked 

Tq: Number of prerequisites directly callable as subproblems; Tp: Total 
number of prerequisites 

The rationale for the parameters of the polynomial was as follows. 
PardTTieter 1 contributes to the weighted sum if the student's level in a concept 
ha:; ju5:t dropped. Obviously, he has not mastered this concept. A decrease in 
I(^vel normally results in more system monitoring and help. Parameter 2 favors 
the concept that has been waiting the longest. Parameter 3 favors a concept that 
in in n new or relatively unstable level range (R « 1). Parameter ^ favors a 
concept whose level is changing rapidly. Parameter 5 favors the concept which 
IS based on the most prerequisite concepts since working this concept will help 
to review earlier concepts. Parameter 6 favors, the concept which currently has the 
lowest level. When a concept has been mastered (level > 3) a large decrement 
is added to that. concept 's score. 

Observation of students * in the course showed that they initially relied 
heavily on the concept selector. As the course progressed,: they assumed more 
; of the task of concept selection. Part of the reason was their\ increased familiar- 
ity with th^ course contents and knowledge of their weaker areas. This aspect 
of the s i tUrit Ion was encouraged as it normally resulted in less waiting time 
between concepts and the students seemed quite proficient at determining their 
own needs. 

lor two semesters, data was collected on all student and system concept 
:.«• it'cl Inn:: (8). A rocord waG kept of parameter values for each considered concept, 
{\iO actual v:t>ncopi reelected, student acceptance or rejection, and subsequent change 
III levf»l fo7- il'irK concept. Fourteen additional parameters were added to the 
oi'if:ii,al six (see Table 5) and a multiple linear regression analysis was performed 
to calcujdLC; optimal coefficients for the twenty independent variables. The 
dependent v..iriable value associated with each concept selection was made proportional 
to iho rcLiultinp, change in level as the goal of concept selection was to advance 
Q the Jitudont as rapidly as possible. 

ERIC 
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TABLE 5: New Independent Variables 

Var . 1-Var 6 same as Table 4 

Var. 7 ALj (last level change) 

Var. 8 WTAVGj (weighted average level change) 

Var. 9 Li (level) 

Var. 10 // of days since last worked concept I 
Var. 11 // of days since last used system 
Var. 12 Total of # of concept worked 
Var. 13 Total // of days worked 
Var. 14 Var. 12/Var. 13 

Var. 15 # of times student worked concept I 
Var. 16 // of times student rejected concept I 
Var. 17 // of times system selected concept I 
Var. 18 // of times student selected concept I 
Var. 19 Var. 16/Var. 17 

Var. 20 of times worked concept I at present level range 

This intermediate step produced a single scoring polynomial which should 
tiK-opptically be better than the original polynomial with only six variables 
arnl non-opl imdl coefficients. To compare"' these polynomials, a second regression 
• in.ilyijis waii j.erformed using only the original six independent variables as 
i>r<-(lictors. Tliis analysis produced a multiple correlation coefficient of .18 
.ir, compared to .48 for the expanded scoring polynomial. 

The findl step was to perform a cluster analysis on the data, collected in 
.in attempt to identify four groups of similar students. The results are shown 
in T.il)]o 6 alon;^ with the multiple correlation coefficient associated with 
each proup. For all except group #3, this coefficient is -higher than that 
ds.sociated with the population as a wheal e (.61, .61i<U3, .53 '.versus .48) and 
should t)iu.s Jead to improved concept selection for the majority bf students. 

TABLE 6: Regressions on Groups 
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r.omo ol.f;ervations froir. this table are that variable 19 appears at or near the 
u>P ni All ^troups. Variables 5 and 9 appear near the top of all groups. Variable 
? l.cis: a relativeiy hi^rh significance in groups 1 and 2 while variable 7 has a 
relatively hlp,li significance in groups 3 and 

The coinjjete concept selection cycle consists of the following steps: 

1. Dtftermine the student's current group 

2. rind the current plateau 

3. Identify candidate concepts 

Evaluate each concept using scoring polynomial for student's group. 

5. I'resent highest scoring concept 

6. Update scoring polynomial compatability ratios 

^tej.:: 2) through 5) have been discussed previously. Steps 1) and 6) are accomplished 
in tlie following manner. A compatability ratio is maintained by student for each 
r.corin^r polynomial. The student is placed i^a the group which has the highest compati- 
hllltv ratio. The concept which would be chosen by each group is determined, and 
Che concept selected by the student ' s current group is presented. After completion, 
the compatibility ratios of all groups which selected this concept are updated to 
refloct the student's change in performance level. In the event the student rejected 
the concept selected and chose another considered concept, the compatability ratios 
of" all fj;roups which chose this concept would be updated instead. 

The next section will discuss results of classroom experience with this 
generative CM system. 

Ki:SUr.TS AND CONCLUSIONS 
It is a rather involved problem to control an educational experiment carefully 
<»noup.h to obtain a meaningful comparison of one teaching method versus another, 
in electing control groups, there are many variables which enter in such as motiva- 
lion arid aptitude for the course, inherent favorable or unfavorable biases towards 
. rmiputers, and personality traits which are difficult to measure. Most of our 
• '111 liu^iasrn and interest in this project has been spurred by conversations with 
::i.wdents who have used this system and an examination of student questionnaires which 
r'atp the sy^'tem very highly as an educational tool (more on this later). 

In any event, an experiment was set up during the fall semester of 1973 in 
which two randomly assigned classes (determined by the unversity class scheduling 
algorithm) of approximately thirty students each were designated as GCAI and control 
Kfoup respectively. Both classes had the same instructor and used the same text- 
book. The basic differences between the course structure for each group are shown 
below: n -t 
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GCAI Class Control Class 

2 hours of lecture/week 3 hours of lecture/week 

homework problems done thru GCAI conventional homework assignments 

seven 20-minute quizzes two 75-minute exams 

The GCAI students were required to take short quizzes after each plateau. 
These were administered on an individual basis and were primarily used to 
• •ncour^dge students to keep a reasonable pace. 

The GCAI system was implemented on an IBM 360/65 computer. Students worked 
at chair own pace and were free to use it whenever the time-sharing system, CPS 
(9), was operating. 

Prior to the start of the course, both groups were given the same pre-test 
which was a simplified version of a previous semester's final exam. The mean 
score for ths control group was 25 out of a possible score of 100; the GCAI 
class mean score was only 9. Median scores for each group were identical to the . 
mean scores. 

rarlier analysis of past student performance in this course had shown the 
single best predictor 'of success to be the student's overall quality point ratio 
(Ol'R) or cumulative average of grades since entering the university. The mean 
OVK for the control group was 2.8 out of a possible 4.0 versus only 2.5 for the 
CCAl jiroup. 

Both groups of students were given the same final exam; however, there was 
no .';ip,nificant difference in their performance (mean score GCAI - 115/150, Control. 
113/150). Since the GCAI groupt spent 1/3 less time in class and was not as well 
prepared for the course, this is a^very favorable result. 

This course is a prerequisite for an advanced digital design course. There 
were approximately ten students from each group in this course during the following 
nemester. Apair. there was no significant difference between the grades received 
by the GCAI students and the control group. 

Three different instructors have used the GCAI system over a period of two 
years. They generally feel that GCAI is beneficial in that it frees them from 
Ji.ivinp. to discuss at length routine problems and techniques. Instead they are 
able to concentrate on more difficult concepts and intuitive approaches to design 
:r. .^tudentr, arc receiving practice in the basic techniques and algorithms through 
GCAI. 

22 
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nentioned, student reaction to GCAI i,s generally quite .^avorable. 
jj or a q " ' " 

TahlT' 7. 



v..» rosuit. Of a questionnaire returned by sixty-four students are sut-ari.ed in 
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TABLE 7: Student Questionnaire 

I preferred this system to conventional homework assignments 
U A SA 

^ 1^^^ m m. 

CAI is an inefficient use of the student's time 
20% 41% 

( was concerned that I might not be understanding the material 
11% 44% 17% 24% 3% 

CAI made it possible for me to learn quickly 

16% 17% 60% 5% 

The CAI system did a good job of selecting concepts 

9% 29% 23% 36% 3% . 

I found the "example mode" feature useful 

0* 3% 9% 50% 38% 

•Strongly disagree; D: Disagree; U: Uncertain; A: Agree; 
* Strongly agree 



The only question which is not responded to in a manner which indicates strong 
..cccptance is ^5 dealing with the goodness of concept selection. Here 39% of 
th<. ntudents responded favorably and 38% unfavorably. We feel this is a reasonable 
..ch,.vement in a very difficult area of decision making where there is clearly no 
-n,i.e right or wrong solution. This is a somewhat better" rating than that received 
Uy the earlier scoring polynomial. 

In conclusion, we feel as do the majority of students who have used GCAI- • 
.t (provides an opportunity to learn by doing and is an effective way to teach' ' 
l.rohLern-oriented material. Students receive immediate feedback and learn from 
the.r mistakes. GCAI helps a student master the basic concepts and enables the 
.-.tudent and instr.uctor to concentrate on more advanced material. , 
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ii. cnKKRATIVE CAI IN MACHINE LANGUAGE PROGRAMMING 
I • Intr'oduc cion 

I>rior reports have described the MALT System (MAchine Language Tutor - 
See References, 5, 10.). MALT is concerned with teaching machine-language program- 
.Mi.:p. for a simplified version of the Digital Equipment Corporation PDP-8 mini- 
computer. The instruction set of the hypothetical machine is virtually identical 
to that of the PDP-8. The major difference is that this computer has only HOO 
m-:...ry registers; consequently, each register is directly accessible. This means 
fb.at -.Ludents can learn the fundamentals of machine- language programming without. 
t.iie added complexity of memory page consideration. 

MALT is a generative CAI system in two important senses. First, it creates 
itv. own sample programming problems using a variety of heuristic techniques. It 
is -not dependent upon the course author for a complete supply of ready-made problems 
and their solutions. Instead, by beginning with only a series of basic problem 
elements or sentences, it generates a problem that is consistent with the user's ' 
present aliility. 

Another important way in which the system is truly generative is its ability 
t.o dei^ign a solution program for the problem that it "has generated. By using 
I'n.'dc algorithms supplied by the course instructor, the system can produce the 
actxx^l niachine code of a solution program. This implies that the system is quite 
( l.e>:il)le, since later alteration and extensions involve only the addition of new 
programming algorithms, not massive system reorganization. 

MALT attempts, through constant monitoring of the student's program to deter- 
mine not only the existence of logical errors, but also their location in the 
p.--ocram. This ability enables the system to be much like the human teacher; that 
i-, iL can note and correct logical errors before they develop into undesirable 
programming, habits. 

The system attempts to tailor its presentation to fit the abilities of the 
students. Any problem that is generated is designed to provide the student with 
a challenge, while not being beyond his capabilities. The dialogue initiated by 
the system will also be governed by the user's performance. A beginning student 
will receive a wide variety of hints and suggestions for the design of his 
program. Also, his errors will result in quite explicit and complete remedial 
messages. Ac the student r... ogresses through the material, he will receive less 
.'••ystem information and be given more freedom in his programming actions. In 
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addition, tije more elementary sub-tasks will be programmed for him. When the student 
.iclii<^vrrr; hlKh prof iciency, the system can function purely as a problem solver in 
tiut all programs will be generated by it. This facility is useful if a student 
desires to study examples of advanced problems and their corresponding solution 
programs. 

A:: the system questions the student, it is constantly developing its own 
iiolution program for comparison with the student's program; In this way, a given 
pro^,ramming concept is rarely presented the same Kay more than once to a particular : 
.student. The student's enjoyment of the system is thereby greatly enhanced because 
he receives new dialogue with every problem. 

The actual operation of MALT is straightforward. After the student identifies 
himself to the system, his records are obtained and evaluated. These records deter- 
mine the difficulty and content of the problem generated and the amount of instructional 
guidance that the student will receive. Next a sample programming problem suited 
to liis abilities is generated. To help him design his program, the system will then 
develop a logic chart or list of sub-tasks. These sub-tasks break the problem into 
a series of smaller, more manageable steps and are of great help to the novice 
programmer. 

As the student undertakes each sub-task in the programming process, a 
corresponding concept routine is entered by the CAI system, which guides the student 
through the construction of that part of his program. During this phase, the student 
is constantly being given feedback as to the correctness of his program. If his 
progrram introduces logical errors, the system will point these out and offer help - 
ful suggestions for their correction. If the system checking procedures determine 
that the student might benefit from observing his program in operation, it also 
has the capability to simulate statement by statement program execution 

Figure 2 is a block diagram of the Malt system. Table 8 provides a complete 
example of the operation of MALT. The comments in square brackets have been added 
to clarify this example for anyone not familiar with the PDP-8 instruction set. 
Ccich student response is preceded by a dash':,* The dialogue shown is that which 
would be received by a beginning student. 
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Figure 2 - MALT Block Diagram 
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TABLE 8: MALT Sample Dialogue 



Your problem is to write a program which wtti: 

Read m 20 (octal; ASCII characten and store them in regbters 232 

through 252. 

Form the sum of registers 232 thru 252 in the accumulator. 

If lAis results in a nonzero LINK, stop with the (ACQ • r,77, 

otherwise stop with (ACQ » O0(X). 



Is this problem OK? 
—yes 

Here are the sub-tasks for the first line. 

1) Initialize a ptr to register 232. 

2) Initialize a cir with the value of -*20(octai). 
Read in a character. 

. Store it away using the ptr. 

§ Update the ptr. 
Update the ctr and if it is not zero. Jump back to start of loop. 



3! 



SUB-TASK 1; / 

Since we may need a ptr later, we will store the ptr value in mcmoiy 

and move it to a register which we will use as the actual ptr. 

Use M377 to store the ptr value, and M376 as the ptr. 

M377: . ^ ^ 

-0232 ^ 

Now, get the ptr value: 

MCOO: [location or 1st instruction] 

- . ^'6 [cdd in contents of M3761 

W DNG! 

'TAD 377" is coded as : 1377 
And deposit it into the ptr: 
MOOl: 

—3376 (Deposit ACC into M376. dear ACC] 
SUB-TASK 2: 

We will store the ctr value in memory and then move it into another 
register for use as the ctr. 

Use register 375 to store the ctr value and register 374 as the ctr. 

M375; 

—7757 

7757 can't be used as the ctr value. Try again! 

Mi75: . . 

—7760 {-20 in 2's complement form] 

First, get the ctr value: 

M002: 

—1375 

And deposit it into the ctr. 

M003: 

—3374 

SUB-TASK 3: 
Tlrst, check the reader flag. 
M004: 
—5031 
NOf 

To check the reader flag is 6031. 

Next, jump back one register if it's not ready. 

M005: 

— 5004 [JUMP to M0O4.] 
Now, operate the reader. 
M006: 
•-6036 

SUB-TASK 4: 
M007: 
—3376 
Wrong! 

•*DCA I PTR*' is :3776 [This should be an indirect deposit.] 

SUB-TASK 5: 
OK, update the ptr. 
MOID: 

—2376 [Increment M376 and skip next instruction if M376 contains 
0000.] 

SUB-TASK 6: 

Now, increment the ctr. 

MOIl: 

—2374 

And if not zero, jump back to start of loop. 

M012: 

—5002 

Wrong! 

The loop starts in register 004 so it is 5004 
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2. Results and Conclusions 

The system has been implemented in the CPS (9) language on the IBM 360/365 
at the University of Connecticut Computer Center- Students can use this system 
whenever tliey desire. There is also a batch mode simulator of this computer 
which they use for class projects of a more ambitious nature. 

l-itudent reaction to MALT has been very favorable. They feel this system 
helpr: to bridge the gap between what they have learned in class, or from the 
tc»>:ti)Ook, and what they need to know to program independently in batch-mode, 
formally, students spent two weeks using MALT and were then given a week to get 
a T*ather sizeable problem coded and running in batch-mode. This proved to be 
sirnificantly easier for students who had used MALT than for those whp had not 
(£e(* question 2 in Table 9). 

A questionnaire was distributed to the classes using MALT. The results of 
this questionnaire are tabulated in Table 9. It appears that the students feel 
that this expo,rience was beneficial and good preparation for learning to program 
independently. On the whole, students were not bothered by the fact that MALT 
requires thern to adhere to a particular "flowchart". As indicated by question seven, 
improvements could be made to the algorithm which determines that a generated 
jirol-Iem is sufficiently different from previous problems presented to that student. 
Ninety students responded to the questionnaire. 
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TABLE 9: MALT Student Evaluation 

Tor riuestion 1-9 the percentage of students giving the following responses are 
tabulated 

Strongly Disagree Uncertain Agree Strongly 

Disagree Agree 

1. The System was useful in introducing me to machine language programming. 

2 ' 2 i| 55 35 

2. Ii was relatively easy to learn to use the batch version of the assembler since 

I had been introduced to programming concepts through MALT 

0 5 18 50 26 

3. Since the sub-tasks were always laid out for me, I felt very constrainted using 
MALT 

5 49 21 25 0 

Because the sub-tasks were laid out, I only learned the mechanics of programming 
and really didn't understand what was going on. 

9 ^5 31 ,11 3 

5. The approach taken in printing out the sub-tasks was good as it taught me how 

to organize a machine-language program. 

0 U 20 62 m- 

6. The problem became more difficult as my level increased. 

2 11 21 60 6 

7. There was a good variety in the problems I received in MALT. 

6 28 17 i|8 1 
a. In p.eneral, I enjoyed the interaction with MALT. 

1 0 li| 69 16 

Overall, ve fdel that MALT is an effective demonstration of what can be 
accomplished in CAI with the limited use of AI techniques. It should be stressed 
that MAbT's desip.n has been influenced by AI research, but certainly much more 
cuuld be done in the way of incorporating AI Research in problem solving and 
r-r'Ograni synthesi::. The desire to produce a working system with reasonable 
i'GSponr.e time on an existing time-sharing system precluded this possibility. 
i:opo fully, MALT will challenge others with an interest in CAI and AI to purse 
r i • podl further. 
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\\-'. COMPnTER ASSISTANCE IN THE DIGITAL LABORATORY 
J . Introduction 

Most of the CAI systems implemented so far teach svibject areas that are 
normally Lein^ taught in a classroom environment. Very little work has been 
clniif^ in the area of laboratory instruction. Neal and Meller [11] have implemented . 
:;v:.ten) which teaches students to operate laboratory electronic instruments. 

In the field of computer science, the student's backgrounds are quite 
v.Iver\se. Many of rhem come from fields other than electrical engineering and 
t!,ey have little or no knowledge of the use of electronic laboratory equipment, 
iiowt^ver, most of them have, learned the basic design tecluiiques for digital 
c:Irc:uitry. - 

It is quite difficult for someone with no knowledge of laboratory instrument- 
ation to implement a circuit design using standard integrated circuits. A computer 
:iy:;tem has been designed and implemented to help them in the construction, debugging 
and testing of a digital circuit. 
2. Overview of CAILD System 

CAILD is a Computer Assisted Instruction system for Logic circuit Debugging 
cind testing. It is implemented on a PDP-9 computer system. (See Figure 3 for 
CAILD Block Diagram). 
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By using the CAILD systein, students are able to wire up their own logic network, 
debug it and test it without any knowledge of the use of laboratory instruments. 
The CAIL. system consists of a Student Logic circuit board, and interface network, 
a monitoring network, a test probe, a Tektronix UOlO display terminal and all 
systems software that is responsible for the instruction of dogic circuit de~ • 
bu;^ging and testing. 

The student designs a circuit and prepares a wiring diagram prior to the 
ti:>e of the system. The student's circuit is wired on the Student Logic Circuit 
board, NAND gates and flip-flops are provided on the circuit board. The student 
merely has to interconnect logic elements on the circuit board to build the des- 
igned circuit. The circuit board is then interfaced to the PDP-9 computer and 
the student inputs the circuit equations into the computer, 

A syntax checker checks for syntax errors made in the equations and the NAND 
format conversion routine converts them into NAND format since only NAND gates 
are available on the circuit board. The debugger in the system compares the out- 
put of the physical circuit with the simulated output of the equations for each 
possible input condition in order to verify whether the physical circuit, in 
fact, realizes the given set of equations. If they do not agree, the s^tem will 
y.uide the student through the whole debugging process until all the *^sources of 
error are found and corrected. See Table 10 for a sample session. 

TABLE 10: CAILD Dialogue 
V/hat is the output circuit point for variable Jl? 

Equation under test: 

Jl=(-Xlf-X2^y3)'^(-XltX2+-y3) + (Xl+-.X2f-y3) + (Xl+X2+Y3); 
Do you wish to print out all test conditions 
(y or N) 

Simulated output condition: 1 
Actual circuit output condition: 0 
Input conditions: XI X2 y3 

10 0 . 
What is the output circuit point for the following term: 
(-Xlf-X2fy3) 
P22 ; • 

Simulated output condition: 1 
Actual circuit output condition: 1 
Input conditions: XI X2 YS 
• 1 0 0 

put the test probe on the input of the next higher level gate which 
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Table 10 - continued 

is connected to the output of (-Xl-h-X2fY3) 
Type alt mode when you are ready. 
(User puts the test probe at the test point and types alt mode ) 

Either you have a wiring error or an open circuit at the test point. 
Please correct the fault. After the correction, type alt mode 
to return to the debugging phase. 

(User corrects the fault and types alt mode) 
Equation under test: 

J1=(-X1 Y3) (-XlfX2f-Y3)f(Xl-h-X2-h-Y3)-h(XlfX2^^Y3); 
Do you wish to print out all the test conditions? 
(Y or N) 

N; 

Testing conipleted for one equation. 
Please tell me the circuit point for variable Kl. 
03U; 

Note: indicates the response for the user 

^^■te^:, debugging the circuit, a test mode is entered. Under this mode 
the student is able to specify the input conditions and present st; te for a 
sequential logic circui-. If the circuit is a combinational circuit, the student 
specifies input conditions only. The system applies these student specified 
conditions to the physical circuit and returns the resulting output and next state 
to the student. By observing the output and next state of the physical circuit, 
the student is able to determine whether the physical circuit realizes the original 
desip^n. 

CAILD by itself is not a complete CAI system since it does not teach 
how to design lop:ic networks. However, CAILD can be use4 in conjunction with a 
CAT system in which students learn the basic concepts of comptlter science. Certain 
concepts in this system are devoted to the design of combinational and sequential 
lo>uc circuits. The design problems are generated by this system. The solution 
to the desif^n problem results in a set of equations which describe the circuit. 
At this point the student can wire up the circuit on the Student Logic Circuit 
board. The circuit will be debugged and tested under the guidance of the CAILD 
systenj. Finally, the student will obtain a working circuit for the original design 
probJr^iii, 

Students can build either combinational or sequential circuits. There 
.irc two set:; of four flip-flops (JK and Delay) available, but only one set can 
bo u:;ed at a time, iience, sequential circuits with up to sixteen states can be 



-26- 

There are two external input lines, 32 NAND gates and twelve inverters. The 
outputs of either set of flip-flops can also be used as external circuit inputs for 
f:onibinational circuits. This allows the design of fairly sizeable combinational 
circuits involving up to six input lines. The number of outputs for combinational 
circuits is limited only by the available logic elements. 

3. System Evaluation 

CAILD has been used in an introductory computer science course which teaches 
ihe design of digital networks as well as programming a minicomputer. There were 
twenty students in this class. Eleven were electrical engineers; the rest were 
predominately mathematics majors. None of the students had any prior exposure 
to digital design. The electrical engineers were taking their first electronics 
laboratory and would encounter experiments on digital logic design later on in the 
semester- The majority of students were sophomores. 

The students in this class were required to design, construct, and debug 
both a combinational and sequential circuit with the aid of CAILD. The majority 
of ntudents were able to successfully complete this task in about three and one- 
half hours (=60 minutes design, =90 minutes wiring, =60 minutes debugging and 
testing with CAILD). Some of the circuits designed were full-adders, code- 
converters, decimal counters, and shift registers. 

All but two of the non-electrical engineers found this to be a very stimula- 
tinr part of the course as it gave them some hands-on experience applying the 
concepts they had learned in class. Similarly, all but two of the electrical 
ongineers found this to be a very helpful preparation for their future project 
in the electronics laboratory. 

The basic system is currently being expanded to include additional sub- 
routines which will enable the student to make use of CAILD prior to actual 
circuit construction. Students will be able to verify that their equations accurately 
r(?present the transition table or truth table they h^d in mind. A wiring diagram 
will 1)6 printed out and saved by CAILD for use during the debugging process. 

Tn summary, we believe CAILD is an effective tool for teaching the use of 
Integrated circuit chips. It also teaches students how to locate faulty com- 
ponents and detect wiring errors. It makes their classroom work more meaningful 
<ir> it provides them with some actual design experience • 

T\\o .ipplicntLon to digital network design is, of course, most natural. However, 
the basic philosophy of CAILD could be used in the computerized teaching of other 
laboratory courses. _ . 
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IV. A GHNKPATIVL APPROACH TO THE STUDY OF PROBLEMS 
1 . Introduction 

This portion of the report describes research on problem solving for generative 
CAT, The goal of this research is a formalism for problem generation which is 
ad.iptable to many different subject areas • Since any implementation for a specific 
problem area will make use of application dependent information and procedures, 
xho ^reneral formalism is intended to provide an approach which can be tailored 
and extended as application requires. 

Following sections include a model for generation, a formalism which provides 
r»omo details of the model, and examples which illustrate the use of the model. 

The model incorporates heuristics which correspond In a dual manner to 
Polya type heuristics for problem solution. The intent of the generation paradigm 
mip.hl well be labeled, how to generate it. The formalism studies the possible " 
::lructure a problem can have in terms of basic problems and the relationship of "■ 
liiu-i structure to that of the solution to the problem. Possible structures of 
a problem are expressed as operations on subproblems. 

lUmristici^ of Problem Solving 

In this study we should not neglect any sort of problem and should find out 
c'cimmon features in the way of handling all sorts of problems; we should aim at 
^'.cneral features, independent of the subject matter of the problem. Moreover, 
"the study of heuristic has 'practical' aims; a better understanding of the mental 
operations typically useful in solving problems" will be directly applicable to 
the teaching and learning process. 

In How to Solve It (12) Polya describes a problem as consisting of three parts: 
tho unknown, data, and condition. He presents corresponding heuristics which can 
holp to find a solution to the problem. Polya presents a general approach to 
solving a problem: 1) understand the problem 2) devise a plan 3) carry out 
the plan H) examine the solution. 

A problem can be understood by identifying its parts (unknown, data, and 
condition) and by-establishing relationships among these parts with respect to 
a base of semantic information. 

A plan for solution may be obtained by using the semantic base to find a 
connection between the data and the unknown, subject to the restraints specified 
by ihe. condition portion of the problem. The plan will make use of subgoals which 
can be derived by using related problems. ■ A subgoal may be part of the original 
j)rolilem, a similar problem having the same unknown, the same problem with auxiliary 
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elements introduced or terms replaced by their definitions. Problems may be 
rnl.iff^d nci^onlinp to subpart, analogy, specialization, or generalization. Related 
iM-.'Mnihrt . nu I n .nrrtvtsU ai vai'yliji^ anU modifying the three parts of a.ppobieni. 
T)ui««; heurit;t.ic guidelines are more of an 'art' than a science; it is an art to 
recognize the utility of a related problem. " 

A plan consists of a sequence of subproblems and subgoals, each of which 
r:liould be solvable, such that the combined solutions give a solution to the 
or .i)^.inal 'problem. A plan directs the synthesis of a solution routine out of 
Mibgoal solutions. 

The third step is the implementation of the plan. Each subgoal of the plan 
must be attained or an alternate found. A plan is a skeleton of a procedure for 
obtaining the solution. A plan must be refined from a general form to a detailed 
procedure. During this progression it may turn out that the plan fails. If so 
it must be patched or discarded. 

The last step is solution verification. The synthesized solution routine 
can, hfi checked by specialization, "generalization, or by variation of the' data. 
Specialization is the application of the solution routine to a special case or 
ruibci.ass of problems; generalization, the application to a more general problem. 
The ::olution should work for special cases and for varied data values and perhaps 

may work for other problems. 

Tlio four steps of 1) understanding 2) planning 3) execution 4) verification 
art' rhemyelves a general plan for a general problem solver. 

2. The Generation Process 

Oeneration is the process of producing many specific items from a general 
object. Generation is dual ('reverse' process) to analysis which is the passage 
from the specific to the general. The specific items are c411ed interpretations; 
I he )umeral object, a representation. 

Formally, the relationship of a representation to Its interpretations is 
dt» fined as a set function from the collection of representations to the collection of 
clasnes of interpretations. This set function and a corresponding representation 
iiavo the same expressive power as the collection of all its interpretations, and, 
inor'f'over, offer the potential of more powerful operations and transformations. 

By making the set function dependent on other variables it can be constrained 
::o map into a part icular subclass of interpretations. For example, it may be 
made dependent on student performance or the modality of a verb; the former, for 
t he f.eneration of problems in CAI and the latter, for the generation of sentences 
having verl)G of a desired modality. 
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f:AI, anrl, in particular, the teaching of problem solving, (U,13) provided 
I ho initial motivation for this investigation. Koffman utilized a concept tree,(l3), 
the arcs of which corresponded to subproblem or prerequisite relationships, as 
a control structure which determined possible calling sequences among the generation 
and Solution routines of his CAI system. 

A. Problem" Generation ' 

For problem generation the representation involves a set of objects and 
pcr.r.ible relationships which hold between them. The set of objects and relation- 
.'•hips is called a problem model. A problem itself is an inquiry which seeks 
properties or consequences of a given interpretation of the model. 

Following Polya, problems have a goal or solution, which may be an unknown 
va]ue, a sequence of actions, a program, a proof, etc. In addition, all problems 
have explicit information, the data, which is given and is usually needed to solve 
a problem. Other informat ion which is implicit may also be needed to solve the 
problem. This implicit information is not part of the data, it may be contained 
in a semantic network. Finally, the goal or solution is determined by the 
condition expressed and any relationships implied by the problem model. 

This paper will deal with quantitative problems as opposed to problems of 
non-numeric reasoning, and the representation used will be a LISP list of the. 
form ((unknown variables) (data variables) (relation) ) where each of the three items 
In the list is a sublist and (relation) is a LISP routine which expresses a 
relationship which holds among the data objects and the unknowns. 

Denote a problem representation by P.. Let denote the class of problems 
: represented by P^. Problem generation, then, is a map which has P^ as a dependent 
v<iriable and which takes values in C. 

The sublists determine problem subclasses. For example, let be the 
collection of force, mass, acceleration problems of elementary physics. Than a 
r.uhclass of is represented by P_j^^ = ( (force)(mass acceleration) (function)) 
whore function expresses the relationship given by the equation force = mass • 
.'icceleration. 

In this context, an implementation of a generation map can involve several 
phanos. In the first phase, the sublists can be manipulated, combined, or 
porciouG of them generated or deleted to produce new lists. A second phase could 
provide a stiiuciatic interpretation of the representation list, using, for example, 
.1 :.»-nwuitic network. A third phase could then express the interpreted list in a 
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TheMrst phase is of primary concern here because the generation of representa- 
tions will provide a significant increase in generative capability. 

3. Duality and Problem Solution 

The solution to a problem is essentially determined by the expressed and 
implied relationships which hold between the data and the unknown items of a 
prol.lem. Indeed, the discovery of these relationships is often the goal in 
•solving a problem. For. example, the function of the representation ((force) 
(mass acceleration) (function)), gives the key to the. subclass of force, mass, 
.acceleration problems represented. 

Thus, in the list representation, the problem itself is represented by 
((unknown) (data)) and the solution by (function). 

Let C be the class of problems under consideration. A problem solving 
system is described by a pair (R, S) where R represents a subclass of C and S 
it: .1 solution method or routine for this subclass. 

Given a collection {(R^, s^)}, the formalism below investigates the following 
.-■;el of interrelated questions: 

1) how can the problem subclasses be extended while keeping the collection 
of solution routines, {S.}, the same 

2) what are some relations on {(R^, S^)}; what structures exist on this 
collection 



3) 



what manipulations and operations can be performed on the lists P. to 

1 

produce new representations 



(duality) what are corresponding operations on the procedures S. which 
produce solution routines for the new classes. 

The duality of problem and solution operations will be expressed by an 
operator pair which operates on the list representations, i.e. the problem operator 
acts on ((unknown) (data)) and the solution operator part acts on (function) , 

The objects (R^, S.) are general and S. could represent special routines, 
nuch as (DEFUN SrORCE' (XY) (SETQ FORCE '(PRODUCT X Y)) where X will be mass and 
Y 'deceleration. could also be a general problem solver such as a theorem 
prover. An appealing use of a general deductive system (such as a theorem prover) 
would be for high level inferences for the manipulation of the (R., S.). Also, 
the deterrrdnation of which to employ and how to use it to solve a problem would 
be decided by some other Sl. 
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An examp],, of a relationship which holds among problems is that of sub- 
problem. The dual relationship on the solution routines is subsolution. This 
structure is rather simple and obvious. Nonetheless, it is important for a 
system to have a way of representing knowledge of such structures. Wiat are other 
structures? The structural information which a generative system uses is know- 
lodRe that a problem solving system should also have. 

4. Abstract Problems 

This following section illustrates the approach to problem generation and 
solution which is used to find answers to the above questions. 

Certain subclasses of a given collection of problems are represented by a • ■ 
structure called a basic abstract problem. Some relations on abstract problems 
are introduced and a partial order on them can be defined. A complex abstract 
problem is one that is constructed from several basic abstract problems. Con- 
siderations of composition techniques using the relations are made. • 

To each basic abstract problem there will correspond a problem solver, or 
solution method. Corresponding '.relationships for solution methods are also 
considered. A complex abstract problem will be-Sdlvable using a combination of 
the problem solvers associated with its basic abstract problem constituents. 

Definitions and Elementary Results 

An abstract problem is defined to be a triple of the form ( (unknown) (data) 
(relation)) where (unknown) is a list of . variables whose'Values are sougfit as " the 
solution to a problem represen-ted , (data) is a list of input variables , and 
(relation) defines a function Vfhich assigns unique values to the unknowns for 
each list of data values. 

An abstract problem which, is input to the first phase of problem generation 
is said to be basic. Basic abstract problems are assumed to be solvable, that is 
have a given associated solution routine. 

for example, ((FORCE) (MASS ACCELERATION) (SFORCE MASS ACCELERATION)) is a 
bar.ic abstract problem. 

An interpretation (un evaluated) of an abstract problem is defined to be an 
association of properties, objects, relationships to the data and the unknown. 
The associations must be se-.nantically meaningful (for example, according to a 
semantic network) with respect to the sublists of the problem representation. 
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For example, an interpretation of the abstract problem given above might 
be the association of a physical object with properties of mass, acceleration, 
and another physical object with the property force, such that the second object 
exerts its force on the first object. A problem in the class represented might be: 
a ball has a mass of x pounds and an acceleration of y feet per second. What 
force Was exerted on it when it was hit by a bat? 

The predicate of an abstract problem ((unknown) (data) (function)) is the 
predicate PR such that PR (unknown data) is true if and only if the value(s) of 
the unknown(R) are the value(s) given by the function defined by (function) when 
evaluated on (data). Predicates can be used for formal proofs concerning abstract 
problems. If the predicate of a problem can be proved from the predicates of 
basic abstract problems, the structure of the solution in terms of basic solution 
procedures is determined by the proof. 

A. Constructions on Abstract Problems 

The constructions on abstract problems involve set theoretic and functional 
operations. Constructions include subproblem, specialization, generalization, 
analogy, transformation, union (or concatenation), composition, cascade, and 
domain dependent constructions. These constructions are implemented as operators 
which apply, on the one hand, to sublists R^ and, on the other, to solution 
routines S^. The main syntactic generative operations are union, composition, 
cascade, and certain kinds of transformations. Subproblem is a decomposition 
technique which yields, ultimately, basic problems. Specialization, generalization, 
and analogy are also special cases of transformation. They have a syntactic 
aspect but also involve unsolved semantic issues. 



A problem generation and solving system is described by a triple (C R S) 
where R is a representation for the class of problems C which can be solved by 
the solution method S. 

The size and variety of C is one measure of the power or capability of such 
a system. Power can be increased by enlarging C while keeping S fixed 2) enlarging 
C while extending S or by 3) decreasing the degree of specialization required of 
the user to implement S while keeping C and S (the method) fixed. 



B. Power 
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The following discussion is primarily directed towards 1) and 3). For 
example, let ( {C^},{R^},{S^}) bena collection where is a subclass of 
problems represnted by and solvable by and the collection is part of a 
CAI system. deals with a -particular concept of the course, may be a 
generative grammar or a generation routine which generates problems dealing with 
the concispt, and solves the generated problems and monitors the student solutions. 
The system can be extended by simply adding more basic elements, (C^, 
Relations on the basic elements could be defined. This introduces a structure 
cn the collection of basic elements, which can be represented by a graph, called 
a concept graph in a CAI application (13). The intent of the relations is to * 
increase the size of C=U^C^ or to decrease the degree of specification required . 
to express solution routines. The relations, in effect, serve as part of the 
control structure. Koffman used the relations of subproblem and prerequisite 
problem and these determined possible calling sequences among the routines {R.} 
and {S^}. The relations will serve as part of the definition of a generation 
map so that a set of elements (or abstract problems) can be represented and ; 
replaced by another smaller set of basic elements and a generation map. Finally, 
• the system could be made more powerful by giving it the capabilities of deter- 
mining the structures on the basic elements or of determining when the relations 
hold rather than having the structures explicitly stored or indicated, say by a 
graph. ■ - 

The basic abstract problems are those which will be used as building blocks 
out of which complex abstract problems, not explicitly represented, can be constructed. 

Further, the basic abstract problems are assumed to be solvable, and, depending 
on the constructions used, the complex abstract problems will also be solvable by 
an appropriate combination of the basic solution routines. We will be interested 
in studying the effect of problem operations on problem generation and problem 
solution routines. 

C. Union 

Let P. and P be two abstract problems with P. = ( (Y^)(X^)(S. )) and 

4 -: J 1 1 

P^ = ((Y^)(X-^)(Sj)) with predicates PR^ and PR^. , respectively. 

^The union or concatenation of P^ and P^. , denoted P^ U P j j is defined to be 
(((Y^)U(Y^)) ((X^)U(X^)) (Sp^y p )) where S is a solution routine for all 
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tho. problems represented and is determined by the conjunction of PR. and PR.. 
'U^, union, on the variables indicates set unions The superscript denotes a"' 
list of variables. 

The union of problems is depicted by the diagram 




p. UP^ 



The implementation of union of problems is denoted by PUNION. 
D, Composition 

Another syntactic type of operation is composition. Composition makes use 
of substitution, which, also, is at the 'heart' of resolution. Moreover, composition 
is fundamental to LISP. 

Suppose P. = ((Y^)(X^)(S.)) and P. = ((Y^')(X^')(S^.)) where Y. is a single element 

and (Y^) ^ (X^). 

Pj • P^, the composition of P^ and P., is defined to be ((Y^*)((X^) U 

((X )-(Y^))) (Sp )) where Sp ^p is determined by the predicate PR.((Y.) 

: ^ 1 i j i ^ ^ 

(X^)) A PR.((Y^')((Y.) U ((X^)-(Y.)))). 

Diagrammatically, P^ • P^ is given by 




(X^)-Y 
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The implementation of composition is called PCOMPOS. 
Cascade 

The cascade operation is a generalization of both the union and composition 
operations. 

Again let P. = ( (Y^)(X^)(S. )) and P. = ( (Y^ )(X^ )(S, ) ) . Also, assume 
that (Y^)n(X-') is not empty. 

P. 0/0 p., the cascade of P. and P., is defined to be (((Y^)U((Y^) - 

^(Y^)n(X^)))) ((X^)U((X^) - ((Y^)a(X^)))) (S p )), where Sp p 

j i j i 

is determined by the predicate PR^ ((Y^)(X^))APRj((Y^")(((Y^)n(X^"))U((X^)- 
((Y^)a(X^))))). 

The cascade of P^ and P^ is illustrated by the following diagram 



P. 

1 



(Yi)rv (x:) 



■»(Y^)-((Y^)n(X^)) 



(X^)-((Y^)A(X^))- 



p. 



->Y^ 



The implementation of cascade is denoted PCASCADE. 

r. Map 

Another natural relation which can hold among abstract problems is that of 

map or transformation. Let P. and P. be as above. 

^ 1 j .... 

A map from P^ to P^ is a pair (f^^, f^) where f^^: (Y^) KY^) and f^i (X^) 36X^) 

such that if PR^((Y^) (X^)) is true then VR^ii^Ul^)) f^{{X^))) is also true. 

Then, if P^ is an abstract problem and f=(fj^, f^) is a pair of functions 
such that PR^( (Y^)(X^)) true implies that ^^^^^^ ((Y^)) f2((X^))) is true, then 

the construction by map gives the abstract problem f(P^) = ((f^((Y'^)))(f2((X'^))) 
(S.)). 

If the condition on the predicates does not hold it will be necessary to 

r.odify S. to obtain a new solution routine. 
1 
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It is possible to define equivalence and partial orders of abstract problems 
and these could be shown to be preserved by a map of abstract problems. 

A simple example of a map for elementary physics is one which changes the 
units of measure. 

The definition of a higher order map would include changes to the solution 
routine in addition to the variables. Thus, a higher order map would be a triple, 
(f, , f^, f^), where f. and f^ are as above and f^ modifies S. such that f^(S.) 
.^i^sipns the proper values to f^((Y^)) for data values of f^CCX^)). 

Some special types of maps are analogy, specialization and generalization* 

An implementation of a map wil] be labeled TRANS. 

5. Solution Routines 

Solution routines for problems resulting from the above constructions are 
determined by predicates obtained from corresponding logical operations on the 
predicates of the operand problems. 

The solution routines for constructed problems can, therefore, be obtained 
from solution operators which correspond to the respective logical operations. 
Theso solution operators operate on basic solution routines to synthesize complex 
iiolution routines. 

Implementations of the solution operators for PUNION, PCOMPOS, AND PCASCADE 
will be termed SUNION, SCOMPOS, AND SCASCADE, respectively. 

Using this notation union of abstract problems, for example, becomes P^ U P.= 

(((Y^)U(Y^)) ((X^)U(X^)) (SUNION S. S.)), 

(SCOMPOS (A) (B) (S)) will mean to substitute A for B in the routine S, A 
.similar notation will be used for SCASCADE. 
TYopositions : 

It is easily seen that PUNION is associative and commutative. PCOMPOS and 
rCASCADE are associative. 

Also, if Pj is a union of abstract problems, then P^ o can be written as 
a union, that is o is left distributive over union. 

Using the basic abstract problems and the operations of union and composition, 
complex problems can be constructed. By the left distributivity of composition 
over union each of these complex problems can be written in a standard form. 

For example, (PCOMPOS (PUNION P^ (PCOMPOS P^ (PUNION P^ P5)))) = 
(rUNlCN (PCOMPOS ?^ P^) (PUNION ( PCOMPOS P^^ (PCOMPOS Pg P^)) (PCOMPOS P^ 
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(PCOMPOS )))). If PUNION is defined as an n-ary operation (a FEXPR in 

Lir.P), the expression becomes (PUNION (PCOMPOS P^ P^) (PCOMPOS P^ (PCOMPOS P^ P^^)) 
(PCOMPOS P (PCOMPOS P. P. )))), which has the form (PUNION P P^P ...) where 
each operand is a basic abstract problem. Similar computations hold for the 
solution operators. 

These results are used to implement SCASCADE. 

Intuitively, SUNION concatenates its argument routines. SCO???OS makes a 
3u{.)Stitution for a variable in a given routine. (SCASCADE (A) (B) (C)) means 
to substitute each element of the list (A) for a corresponding variable in the 
iiiit. (B) for each occurrence of that variable in the routine C. Then SCASCADE 
t:an be implemented by first putting the complex solution of (in o/o p_^) 
in standard form and repeatedly applying SCOMPOS until all the sxibstitutions 
have been made. 

. Control 

The control which specifies the manner in which abstract problems are combined 
and how a solution routine is synthesized is contained within the operators 
themselves. The decision when to apply the operators and which problem operands 
to be operated on is made by a planning routine. For some applications, the 
planning routine need only consist of a random selection mechanism for selecting 
operators and problem representations to be operated on. For instance, for 
problem generation for CAI, such decisions could be made randomly, with constraint 
by a complexity measure. 

For other applications and other operators, the planning routine will need 
semantic information. In these situations the planning routine could incorporate 
path tracing and pattern matching mechanisms for tracing within a semantic network. 

There is an analogue with programming languages. Here the concern is that 
of combining lists and procedures using operators via a planning routine rather 
than combining elementary statements via a program. Also, here the attempt is 
to incorporate decision making capability within the planning routine. 

7. Computational Issues 

Given n basic abstract problems, the number of meaningful abstract problems 
which can be generated using PUNION, PCOMPOS, AND P'CASCADE can easily be seen to 
bcundod below by n and above by 2^-1. 

Thii; does not say that the number of meanir-gful representations generated 
will .ipprojch the upper bound. The actual number which can be generated depends 
ori t:;e mutual relevance of the given n basic representations. For 2AI applications 
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most useful problems will be the result of at most 3 cr H applications of the 
operators . 

On the other hand recall that the discussion still pertains to the represent- 
ation level and each representation will be used in conjunction with, say, a 
semantic net to produce many specific problems. 
Another example 

The above operators are natural ones for quantitative siibjects and for use 
in LISP. 

Suppose a list of basic abstract problems from the topic of uniformly 
.icceleration motion consists of the following three representations: 
1) ((DISTANCE) (AV--VEL TIME) (SDIS AV-VEL TIME)) where distance, average velocity, 
and time are written as LISP atoms which point to a semantic net and SDIS is a 
LUiP function which returns a value for distance given values for average velocity 
. md t [me . 

7) ((ACC) (VEL-INIT-VEL TIME) (SACC VEL INIT-VEL TIME)) where SACC returns a 
value for ACC given velocity , initial velocity, and time as arguments. 
3) ((AV--VEL) (VEL INIT-VEL) (SAVVEL VEL INIT-VEL) ) where SAWEL is a LISP 
routine which defines average velocity. 

I'COMon applies to 3) and 1) gives ((DISTANCE) (VEL INIT-VEL TIME) (SCOMPOS 
(SAVVn VEL INIT-VEL) (AV-VEL) (SDIS AV-VEL TIME))), where SCOMPOS substitutes 
ihe expression (SAVVEL VEL INIT-VEL) for AV-VEL in the function SDIS before it 
IV. evaluated. 

Transforming 2) and composing with the representation just generated would 
r.ivp ((DISTANCE) (INIT-VEL TIME ACC) (SCOMPOS (TRAN VEL (SACC VEL INIT-VEL TIME) ) 
(vnL) (nCOMPOG (SAVVEL VEL INIT-VEL) (AV-VEL) (SDIS AV -VEL TIME)))). 

THAN would be a symbolic manipulation routine for solving for velocity in 
tftnn: of acceleration, initial velocity, and time given (SACC VEL INIT-VEL TIME). 

Tb.c resulting abstract problem has a solution routine which corresponds to 
.1 derivation of the equation distance = (initial-velocity • time) + (acceleration . 
rinie")/2. 

ti. Conclusions 

The paper has represented a r.odel and approach to problem generation and 
::Glulion, which consists of an expiossion duality between problem generation and 
problcr. isolution, a study of the structure of a problem iii terms of basic problems, 
t ho cxprosciion of structural relations as operators and routines, the elevation 
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of some control to a higher level (to a structure on abstract problems and 
solution routines rather than one on elementary programming statements). This 
approach seems to allow easy incorporation of direction and semantic knowledge, 
larjp.e inference, steps, and additional inference operators. It is related to 
lormal lop,ic via the predicate of an abstract problem and has significance for 
automcitic synthesis of programs because it discusses the synthesis of routines 
our of routines rather than elementary statements; for algebraic manipulation, 

'ace it suggests a flexible notation and approach with the possibility of know- 
lod^e manipulation. 

The model and development provide a context or setting which brings forth 
many additional questions such as the incorporation of static knowledge to 
<»xtend some of the operators beyond their formal manipulation capability, 
(Questions on the interplay of static knowledge in a data base or net and dynamic 
knowledge in the form of routines, 'surgery' for the modification of a routine, 
and applications to other problem areas including prograonming languages. 



46 



V. A NATURAL LANGUAGE LISP TUTOR 
1. Introduction 

Natural language understanding is becoming more important in CAI systems. 
Many powerful parsers have been developed for very extensive subsets of English 
including Wood's parser (14) and Winograd's PROGRAMMAR (15). These methods are 
hoRinning to be of use for question answering sub-systems and for interpreting 
student responses. Of particular interest is the work done by Brown et al. (16). 
It is his semantically driven fuzzy grammar that has influenced this research. 

Brown analyzed student scenarios involving electronic circuit trouble- 
shooting and discovered that the concepts and items discussed fell into small 
numbers of semantic groupings or categories. 

These included: measurements, control settings, circuit elements, and 
circuit locations relative to the electronic circuit. The parse was then 
accomplished by anticipating and looking for phrases which matched these semantic 
categories where they might be appropriate. The parser's fuzziness stems from 
the fact that words that are not understood can be skipped over. 

Teaching a programming language embraces two major tasks. One is presenting 

new material in an orderly controlled fashion. The other involves making available 

a friendly, informative environment that allows experimentation, and that can 

reco^\nize errors and misconceptions and point them out. With ISLE (Interactive 

Student LISP Environment) we have concentrated on this second item by developing 

a computer based environment for LISP. The system knows about the structure and 

fioniantics of the LISP language and can answer questions about LISP data structures, 

concepts and functions; both those functions defined by the student and the primitives 

internal to the system. 

As an example consider the following dialogue: 

STUDENT: Is Atom a functior . 

MACHINE: Yes, Atom is a subr. 

STUDENT: How many arguments does it take? 

MACHINE: Atom is a subr and takes 1 argument. 

Notice that questions are answered with a complete statement to avoid 

any ambiguity, also note the use of the pronoun it by the student. Some limited 

anaphoric reference is permitted by the system to allow a more natural dialogue. 
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TTUDENT: Describe it 

MACHINE: Atom is a subr. and a predicate function 

*It returns true (the Atom T) if its argument 
is an atom, nil otherwise. 



STUDENT: What is an Atom? 

MACHINE: An Atom is the most elementary type of s-expression. 
It may be either a number or a literal atom. 

Some words like atom can refer to more than one thing. The correct reference 

is found by looking at the context of the word and making a decision based on 

p Jural markers, determiners, or qualifying adjectives. 

STUDENT: Are property lists kinds of lists? : 

MACHINE: A property or* P list is an alternating list of literal 
atom property names and coi^responding property values. 
It can be search using the function GET. 

Subset-superset and part-part of relationships are used extensively 

by the system for answering questions of this type. A yes answer is implied 

\\o.re by giving • the student a definition of the subset item. . 

STUDENT: Define (((FACT (LAMBDA (X) (COND 

((ZEROO X) 1) 

(T (TIMES X (FACT (SUB 1 X))))))))))) 

MACHINE: Arguments for Evalquote: 
DEFINE 

(((FACT (LAMBDA (X) (COND( (ZEROO X) L) (T(TIMES X 
(FACT (SUBl X))))))))) 
VALUE IS: 
(FACT) 

Any, student input that has the form of a function followed by an argument 

Hf;t is aiisurned to be input for the evaluator. Here the student defines a 

junction to compute factorials. 

STUDENT: FACT (3)) 
.MACHINE: Arguments for Evalquote: 
FACT 
(3) 

VALUE IS: 

Undefined function found. 

It is ZEFOO found in the function fact. 

In defining FACT the student mistyped ZEROP. The error is found during 

ovjluation and returned to the student. Notice that terminating symbols are 

necessary at the end of each student input line. They may be periods, question 

m4irks or an extra right parenthesis. 

' STUDENT: PRINT FACT. 

MACHINE: (LAMBDA (X) (COND & &)) 
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When printing function definitions only the highest elements of the 
lief init ion appear; lower level structures are replaced with 6. 
r.TUDENT: 3 2 p. 

maciiinf:: ( (ZEROO X) 1) . 

Now u:: In)', conimands to the editor, the student looks at the lower level 

sh^rf^ ho T'oiiicMnbers ZEROP should go. 

f;TlJDi::NT: 1 (1 ZEROP) P. 
MACHINE: (ZEROP X) 

The error is now corrected and this can be checked by evaluation or 
hy cuikinp. another question. 

LiTUDENT: Does FACT call any undefined functions? 
MACHINE: FACT calls no undefined functions. 

flTUDENT: FACT (3)) 
. MACHINE: Arguments for Evalquote. 
FACT 
(3) 

VALUE IS: 
5 

:*.y:.:tem Orp.tinization 

The system is built around several submodules as shown in Figure Each 
:;tudnnt command or request is received by the monitor where it is classified as 
input for either the editor, the evaluator, or the parser. Any input which is not 
♦ '.Ii iier a list of edit commands or a function followed by an argument list is 
,i;.:;um<Hl to bo a request for information and so is passed to the parser. Anyirhing 
r he i>driifir can not handle is rejected and the student is asked to rephrase or 
i'l'OtiK up the question. 

The prirser and the semantic routines work together to answer factual 
i\iHKA ion:; about the LISP language, its data structures and functions, and about 
Wiv (unctions that student has defined. The parser mechanism has a semantic basis 
In tluit specific words or phrases' are looked for that refer to things the system 
Inowr- about, i.e. functions, definitions, or entries in the semantic networks. 
'i'h<*::o itemr. are divided into groups or categories that are semantically similar. 

nemant Ically similar items are those that might fit in a given slot in a 
:;Mnt(»nce or question, and that fall into a superset classification such as data 
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structures or function names. The result of the parse is an executable LISP 
function whose evaluation causes a response to be generated for the student. 

The evaluator evaluates student functions when called upon, accepting nearly 
any LISP 1.5 constructions. When student errors are found, it reports the type 
of error and in what function it occurred to the student. Editor commands can 
laen be used to look around inside of function definitions and to insert, delete, 
and change parts of the definition. 

THE GRAMMAR AND ITS IMPLEMENTATION 

The heart of the English understanding component of the system is a 
BNF grammar. After a line has been read in, an interpretation of it is attempted 
using an implementation of the grammar shown in Appendix B. In the SOPHIE system [16], 
every non-terminal is considered a semantic entity to be searched for when necessary. 
In the ISLE system, however, only a few of the rules Sre actually concerned with 
semantic entities or categories. These semantic entities are defined as only those 
things which have entry in the semantic network. The rules which embody certain 
semantic groups have already been described. The rest of the grammar rules are 
used to identify requests for certain relationships or properties of the semantic 
entities. 

Most programs which make use of a grammar use some kind of parser or 
grammar interpreter. This parser (a program) then uses a table or array in which' 
the grammar rules are stored (data). Special control structures must be set up 
to control backing up when an incorrect parse is begun. In ISLE, this grammar 
iz implemented directly in LISP. For each rule (non- terminal) in the grammar, 
there is a corresponding LISP function with the same name implementing that rule. 
The LISP control structures make this implementation relatively easy due to the 
recursive definition of LISP functions in general and the use of the special built-in 
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functions; COND, AND 5 and OR. Backup is automatic as each rule-function can 
let its calling rule- functions know of its failure on return. All pointers and 
variable values will again be those originally set in the calling function. There 
is nothing to undo or redo as the LISP control structure handles this automatically. 

THE SEMANTIC ROUTINES 

The parsing operation, if it is successful will produce another LISP 
function to be evaluated. Some of these functions and the sentences that produced 
them are given in TABLE 11. Each is a call to a predefined semantic routine. 
The functions FN, FTYPE, CONCEPT, and STRUCTURE retrieve the desired semantic 
information for their arguments. In this way words such as ATOM are disambiguated. 
For example, (FN ATOM) will retrieve information relevant to the function ATOM, 
while (STRUCTURE ATOM) will retrieve the information concerning the structure. 
PREP is used to find semantic information for pronouns which it does by matching 
its arguments against the semantic categories of previously mentioned items. 

TABLE 11 Sentences and Their Translation Into 
LISP Functions 

IS ATOM A FUNCTION? 

(RELATE (FN ATOM) (FTYPE FUNCTION)) 

HOW MANY ARGUMENTS DOES IT TAKE? 

(ARGCOUNT (LIST (PREF FN))) 
• DESCRIBE IT. 

(DESCRIBE (LIST (PREF FN FTYPE CONCEPT STRUCURE))) 
WHAT IS AN ATOM? 

(DESCRIBE (LIST (STRUCTURE ATOM))) 
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ISLE's semantic routines are all specialists for answering their own 
. types of questions. Some take information directly from the network to be 
given to the student or to be used in comparison or relationship tests. 
DESCRIBE, for example, gives the student a pre-defined definition or description 
if it exists. In the case of student defined functions, it tells the student 
the type of function it is. RELATE reports on 'superset' * subset', and 'part -of 
relationships between its arguments. 'ARGCOUNT checks the semantic information 
arisociated with its argument, or in the case of student defined functions-the 
actual function definition, to tell how many arguments a particular function has. 

The permanent semantic information used by these functions is set up as 
association lists of relationships and values for each semantic entity. Table 12 
shows this information for the structural item atom. The value of the relationship 
TEST is the name of a predicate function which tests for the associated semantic 
entity. In this case, the function ATOM tests for the structure which is an atom. 
TYF^E and TYPE OF indicate subset and superset relationships, and DESCRIPTION in- 
dicates a literal definition of the item. 

TABLE 12 Semantic Information for the Structure ATOM 
((TEST • ATOM) 

(TYPE or . ( S-EXPRESSION INDICATOR)) 
(TYPE • (LITER.<\L NUMBER)) 

(PART OF • (S-EXPRESSION DOTTED-PAIR LIST))' 

(DESCRIPTION • ((AN ATOM IS THE MOST ELEMENTARY TYPE OF 
S-EXPRESSION (DOT)) 

(IT MAY BE EITHER A NUMBER OR A LITERAL ATOM (DOT))))) 

Other temporary information that might be used by the semantic routines 
can be created and changed in various ways. When a student defines a function, 
the function is analyzed and lists of the variables it binds or uses and the 
Functions it calls are created. This information is used by the routines which 
handle questions about the student's functions and is updated whenever a function 
is edited or redefined. The editor and the evaluator also store information that 
could' be used by the quest ion- answering system. This is done whenever errors 
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occur and includes information about the current state of the evaluator ot» editor 
(e.g. the aarjociation list) and the cause of the error. This would allow the 
student to obtain more information about the source of the error and what the 
evaluator (or editor) was doing before the error occurred. 

CONCLUSION 

This system is undergoing continued development. The question-answer is 
being expanded to allow the student to get more of the information he or she might 
want and to perform more edit functions in English. 

ISLE is implemented in LISP which runs interactively on an IBM 360/65. 
This interactive LISP is an improved version of the Waterloo LISP which uses a 
cathode-ray display as the active user terminal. 

Preliminary indications are that the system will serve as a useful tool for 
familiarizing a student with LISP concepts. The question answering capability 
allows a student to inquire about the semantics of LISP; he can use the LISP student 
evaluator to test his knowledge of LISP sjmtax and to help him correct his errors. ^ 
The expanded diagnostic information presented should help him clear-up initial 
misconceptions and ease his transition from ISLE to the standard LISP evaluator. 

This approach appears to be general in that one could present any material 
of a factual nature in a similar manner. SOPHIE [16] is an example of a similar 
system for teaching electronic-circuit analysis and trouble-shooting. Other 
programming languages, logic circuit design, and basic algebra and calculus might 
possibly be taught using a similar computer environment. " 
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VT OVERALL EVALUATION 

This project has studied the topics of student modelling, concept selection, 
r.^^nerative CAI, problem generation and solution, and natural language in CAI. 
Aii a, vehicle for the study of these topics, several experimental teaching systems 
have been designed and implemented. 

An introductory course in digital design and programming which utilizes the 
f;y5)tems described in Sections I, II and III of tfiis report has been taught to five 
sections of students. ( approxiTnately twenty-five students/section). The results 
of this experiment have been described above and were generally very satisfactory. 

The conclusion from this experiment is that geneoative CAI is a very effective 
medium for teaching quantitative college-level courses. The major advantages are 
the p,uided direction in the problem-solving process provided to beginning students, 
and the presentation of instantaneous feedback and remedials when the student goes 
iistray. Generative CAI also frees the instructor from discussing- routine,, algorltlimic 
procedures in class, and allows him to concentrate on more complex concepts. Through 
the use of a monitor which makes " intelligent decisions concerning the concept 
to be studied, the difficulty of the problem to be generated, and the degree of 
explanation and student monitoring provided; instruction can become Righly individual- 
ized. In this way, the system makes the maximum use of each computer session. 
Tiie student is also free to proceed at his own pace and override any of the monitor's 
decisions. 

The major drawback to generative CAI is the cost of designing and operating 
these systems. The design was, of course, a one-time expense and entailed many 
man-hours of effort. Unfortunately, there is no accurate accounting available 
of the time spent in the implementation of this system. Many routines were written 
hy the principal investigator; some were implemented as portions of Master's 
Thesis; others were independent study projects. 

The cost of operating the system averages out to approximately $100 per 
student per semester. This provides on the average of four terminal hours per 
student per week. Certainly, this is expensive compared to the design goal (not 
■yiS'tr^attained) of the PLATO system of $.50 per terminal hour per student. Unlike 
PLATO, it should be noted that this system was not implemented on hardware 
configured and intended exclusively for CAI; rather, it was implemented on the 
IBM 360/65 computer which was not really designed for time-sharing. 
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To provide the same degree of individualized instruction for twenty-five 
students would require hiring five graduate assistants (assuming a 20 hour /week 
workload). The cost of five graduate assistants for a semester would be approxi- 
mately $8,000. The coat of generative CAI" (approxliimt^^ 
ably less. 

Additional studies in the use of Artificial Intelligence techniques have 
provided a general model for the generation and solution of problems (Section 
IV). This model also appears to have applicability in areas of artificial 
Intelligence research such as Problem Solving and Program Synthesis. 

A hardware-software system was constructed for use in the digital laboratory. 
This system was very successful in teaching students how to apply classroom 
concepts in the design of actual digital circuits. 

In addition to the system described in Section III, a digital services 
system was built which acted as a "front-end" for the original system. The digital- 
services system aided students in all phases of the design problem and subsequently 
transferred control to the debugging phase of the original system to verify the 
cor^ectnegt; of the student's design. Any hardware bugs or faults were located, 
with student ass^istance, and eliminated. 

nodel for a natural language CAI system previously proposed by Brown (16) 
w.^i: adapted for use in an Interactive Student-oriented LISP Environment (ISLE). 
Tiu! implemeritation of this system was relatively straight-forward once the LISP 
Inrerprt^er v*as modified to run interactively on the IBM 360/65. Its use verified 
i')uit AI research 5n natural- language processing could be effectively applied in 
I he desijrj^ of teaching systems . 

On§ interesting feature of all of the research decribed herein is the broad 
spectrum of teaching activities and techniques covered. The activities range * 
from electrical engineering courses in the theory, design, and laboratory construction 
of digital circuits to computer science courses in machine-language programming 
and the LISP computer language. Contributions were also made to problem generation 
*ind solution in general as well as problem-generation in high school algebra , (5) . 
Tho computerized teaching techniques included generative CAI, which was normally 
under system control, a digital circuit debugger which provided a mixture of 
r^tudent and system control, and finally a LISP learning environment which was 
entirely under student control 

56 



-50- 

This research is a small step in the application of Artificial Intelligence 
techniques to CAI. It is expected that future research along these lines will 
contribute to the continued growth of Artificial Intelligence and provide signi- 
ficant enhancement to the intelligent use of ' computers in instruction. 
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EXAMPLES OF PROBLEM GENERATION 

. The following illustration uses nine basic problems from the area of 
elementary physics. Problem representations are generated at random. In order 
to reduce duplication generated abstract problems were hashed using the MACLISP 
system function SXHASH and tfte number values stored. Repetitions are still possible 
since, for example, (SXHASH ' (A B)) ^ (SXHASH A)), yet (A B) = (B A) as sets. 
Hov'ever, the amount of repetition is drastically reduced. 

For this session the solution operators were executed to produce the 
solution routines. All of tKe generated routines have no lambda variables or 
prog variables; all the variables are global to the session. 

Note problem representation ^. It was decided to allow the union of two 
problems. where the input of one is the output of the other. The intent is that 
on an intepreted level this will take the form of two different instances of the 

same variable (e.g. an av-vel, and an av-vel^). 

X 2 

Twenty-three distinct problems and their solution routines were generated 
before the session was terminated by typing in NIL. Problem 23 is a nice example. 
n0016 means that 16 ^DEFUN* rotuines were generated. 

Notice that the third basic problem was not output; the PLAN routine ueea 
ti random or nondeteniinistic control structure. 

Complexity refers to the number of operators applied. It gives one measure 
of the difficulty of a problem and provides one way of defining a partial order. 

(MAIN) 

(ENTER OPERATOR NAME) PUNION 
(ENTER OPERATOR NAME) PCOMPOS 
(ENTER OPERATOR NAME) PCASCADE 
(ENTER OPERATOR NAME) NIL 

(ENTER PROBLEM TUPLES) ( (FORCE)(MASS ACCEL) (SFORCE MASS ACCEL)) 

(ENTER PROBLEM TUPLES) ((ACCEL) (VELI VEL 2 .TIME)(SACCEL VELl VEL2 TIME)} 

(ENTER PROBLEM TUPLES) ( (AV-VEL) (VELl VEL2)(SAV-VEL VELl VEL2)) 

(CNTER PROBLEM TUPLES) (DIST)( AV-VEL TIME)(SDIST AV-VEL TIME)) 

(ENTER PROBLEM TUPLES ( (ACCEL )(GRAV)(SEQ ACCEL GRAV)) 

(ENTER PROBLEM TUPLES )(( WEIGHT) (MASS GRAV)( SWEIGHT MASS GRAV)) 

(ENTER PROBLEM TUPLES) ((MOMENTUM) (MASS VEL)( SMOMENrjM MASS VEL)) 

( ENTER PROBLEM TUPLES ) ( ( IMPULSE) (FORCE TIME) (SIMPULSE FORCE TIME) ) 

(ENTER PROBLEM TUPLES) NIL 

(ENTER GO OR NIL) GO 

(ENTER COMPLEXITY) -2 

iJMKNOWN: ^ 
(ACCEL) &y 

IJATA: , 
(GRAV) 
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SOLUTION: 

(SEQ ACCEL GRAV) 

(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
UNKNOWN 

2(CENTRI-ACCEL) 
DATA: 

(VEL RADIUS) 
SOT.UTION: 

(SCENTRI-ACCEL VEL RADIUS) 

(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
UNKNOWN: 

3(AV-V£L MOMENTUM) 
DATA: 

(VELl VEL2 MASS VEL) 
SOLUTION: 

(DEFUN COOOl. NIL (PROG NIL (SAV-VEL VELl VEL2) (SMOMENTUM MASS VEL))) 

(ENTER GO OR KILL) GO 
(ENTER COMPLEXITY) 2 
UNKNOWN: 

4(niST ACCEL AV-VEL) 
DATA: 

(TIME VELl VEL2) 
r.CLUTION: 

(DEFUN G0002 \'IL (PROG NIL (SDIST AV-VEL TIME) (SACCEL VELl VEL2 TIME) (SAV-VEL VELl VEL2))) 

(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
UNKNOWN: 
5( ACCEL) 

DATA: • 
(VELl VEL2 TIME) 

r.OLUTION: 

(SACCEL VELl VEL2 TIME) 
(ENTER GO OR NIL) GO 

(ENTER COMPLEXITY) 2 duplicate problem generated 

(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
CUNKNOWN: ' 
(CENTRI-ACCEL FORCE) 
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DATA: 

(yi,u j'Aiiiur: ci^av masc) 

oOLUTlON: 

(DEFUN G0003 NIL (PROG NIL (SCENTRI-ACCEL VEL RADIUS) ( SFORCE MASS (SEQ ACCEL GRAV)))) 

(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
CNTER TO OR NIL) GO 
(ENTER COMPLEXITY) 2 
li'NKNOWN : 

7 (MOMENTUM IMPULSE) 

- HATA: 
(MASS VEL FORCE TIME) 

SOLUTION: ' 

(l)LTUN G0004 NIL (PROG NIL (SMOMENTUM MASS VEL) (SIMPULSE FOPCE TIME))) 

(ENTER GO OR NIL) GO 

(ENTER COMPLEXITY) 2 

UNKNOWN: 

8 (ACCEL WEIGHT) 

DATA: 

(Vr:Ll VEL 2 TIME MASS GRAV) 
SOLUTION: 

(nnrUN G0005 nil (prog nil (SACCEL VELI VEL2 TIME) (SWEIGHT MASS GRAV))) 

(ENTER GO OR NIL) GO ' 

(ENTER COMPLEXITY) 2 

(ENTER GO OR NIL) GO 

n-HTER COMPLEXITY) 2 - 

UNKNOWN: 

'KDIST ACGSL) 

DATA: 

(AV-VEL VELI VEL2 TIME) 
SOLUTION: 

(DEFUN G0006 NIL (PROG NIL (SDIST AV-VEL TIME) (SACCEL VELI VEL2 TIME))) 

(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
UNKNOWN: 
10( IMPULSE) 

DATA: • t 

(FORCE TIME) 

SOLUTION: 

(DEFUN G0007 NIL (PROG NIL (SIMPULSE FORCE TIME))) 
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(liNTtH GO OR Nil,) GO 
■ (LNTER COMPLKXITY) 2 

(ENTER GO OR NIL) GO 

(ENTER COMPLEXITY) 2 
• UNKNOWN : 

IKACCEL DIST) 

DATA : 

(GRAV AV-VEL TIME) 
:;0LUTI0N: 

(DEFUN G0008 NIL (PROG NIL (SEQ ACCEL GRAV) (SDIST AV-VEL TIME))) 

(liNTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 

(ENTER GO OR NIL) GO . ' 

(ENTER COMPLEXITY) 2 

(ENTER GO OR NIL) GO 

(ENTER COMPLEXITY) 3 

i.2UNKN0WN: 

(CENTRI-ACCEL ACCEL WEIGHT) 
DATA: 

(V EL RADIUS MASS GRAV) 
:;0LUTI0N: 

(DEFUN G0009 NIL (PROG NIL ( SCENTRI-ACCEL VEL RADIUS) (SEQ ACCEL GRAV)(SWEIGHT MASS GRAV))) 

(ENTER GO OR NILL) GO 3 
(HNTER COMPLEXITY) 3 
13 UUKf.'OWN: 

(DIST FORCE CENTRI-ACCEL) 
DATA : 

(AV VZi TIME MASS ACCEL VEL RADIUS) 
?.or,UTION-: 

(DEFUN GOOlO NIL (PROG NIL (SDIST AV-VEL TIME) (SFORCE MASS ACCEL) (SCEN TRI-ACCEL VEL 
RADIUS))) 

(HNTER GO OR NIL) 
(TRY AGAIIi) 

(i:ntf:r Go or nii.) go 

(i;NTER COMPLEXITY) 3 
UNKNOWN: 
. CwriSHT CENTRI-ACCEL) 

DATA: 

(MASS GRAV VEL RADIUS) 
IJOLUTION: 

_ (DEFUN GOOll NIL (PROG NIL (SWEIGHT MASS GRAV) (SCENTRI-ACCEL VEL RADIUS))) 
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(f:ntkr go or nil) go 

(I:NTER COMPLEXITY) 3 
DNKNOWN: 

isd'ORCF.:) 

HATA: 

(MAr.IJ ACCEL) 
::0I.UTJ0N: 

(srORCE MASS ACCEL) 

(i.NTER GO OR NIL) GO 
(KNTfiH COMPLEXITY) 2 
UNKNOWN: 

.!(.;( FORCE ACCEL IMPULSE) 
DATA: 

(MASS VELl VEL2 TIME) 
SOLUTION: 

(DEEUN G0012 NIL (PROG NIL ( SFORCE MASS ACCEL)(SACCEL VELl VEL2 TIME) (SIMPULSE FORCE 
TIME))) 

(VmV.K GO OR NIL) GO 
(i INTER COMPLEXITY) 2 : 
UNKNOWN: 

17( IMPULSE ACCEL) 



DATA: 

(MASi; VELl VEL2 TIME) 
.^:;OLUTION: 

(DCrUN G0013 NIL (PROG NIL (SIMPULSE (SFORCE MASS ACCEL) TIME) (SACCEL VELl VEL2 TIME))) 

(ENTEH GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
UNKNOWN: 

18(niST) ' ^ 
DATA: 

(AV-VEL TIME) 
I'.oiAITION: 

(:;Dir>T AV-VEL TIME) 
(l.'NTER GO OR NIL) GO 

(i:nter complexity) i 

(ENTER GO OR NIL) GO 

(ENTER complexity) 1 

UNKNOWN: 

I •)( MOMENTUM) 




DATA: 

(MASS VEL) 
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SOLUTION: 

(SMOMENTUM MASS VEL) 



(ilNTHR GO OR NIL) GO 

(i:nter complexity) i 

UNKNOWN: 
20( FORCE) 

DATA : 

(r,RAV MASS) 

:.:oLnTlON: 

(SrOKCE MASS (SEQ ACCEL GRAY)) 

(F:NTER GO OR NIL) GO 
(f INTER COMPLEXITY) 1 
(riNTiiR GO OR NIL) GO 

(i:nter complexity) i 

(ilNTER GO^ OR NIL) GO 
(ilNTER COMPLEXITY) 2 
21UNKN0V.Tq: 

( IMPULSE FORCE) . 
DATA: 

(V1:l1 VEL2 time MASS) 
SOLUTION: 

(HLTUN G00012 NIL (PROG NIL ( SIMPULSE FORCE 

(i:nti:k go or nil) go 
(mntlr complexity )2 
(i.:nti:r no or nil) go 

i:NTi:R COMPLEXITY) 2 
UNKNOWN: 

'^KCLNTRI-ACCEL WEIGHT) 
DATA: 

(vf:-l radius mass gray) 

r.OLUTION: 

iumm Gooib ::iL (prog nil (scentri-accel ye 

(flNTER GO OR NIL) GO 
(1:NTER COMPLEXITY) 2 
-(ENTER GO OR NIL) GO 
(ENTER COMPLEXITY) 2 
(ilNTER GO OR NIL) GO 
(ENTER C0M1>LEXITY) 2 
(ENTER GO OR NIL) GO 
(ENTilR- COMPLEXITY) 3 
(ENTiiK GO OR NIL) GO 
(ENTER COMPLEXITY) 3 
UNKNOWN: 
22 (WEIGHT) 



TIME)(SFORCE MASS (SACCEL VELl YEL2 TIME)))) 



RADIUS )(SWEIGHT MASS GRAY ))) 
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1»ATA: ' • 

(ma:.:. r.HAV) 

S(}LUT10N: 

(SWLIGMT MASS GRAY) 

(DNTr.R GO OR NIL) GO 
(fJNTER COMPLEXITY) 3 
UNKNOWN: 

22( MOMENTUM IMPULSE) 
IVTA: 

(V::l1 VEL2 MASS VEL TIME) 
SOLUTION: 

i\)\:\VU G00i5 NIL (PROG NIL (SMOMENTUM MASS VEL)(SIMPULSE (SFORCE MASS (SACCEL VELi 
Vi:L2 TIME)) TIME))) 

0:NTI:R GO OR. NILl.NIL . 

V 

.K/r ■ . 

k i Io-core-sec=860 - 
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APPENDIX B 
ISLE'S BNF GRAMMAR 
INPUT : : =<EDIT /<EVAL>/<REQUEST> 
-•1:DIT>:: = any valid string of LISP edit commands 

•i.:VAL>;: = any valid pair of items of the form function arglist for evalquote. 

-Kh:QUEST>:: = <DEFINE/Q>/<RELATION/Q>/ TEST/Q>/<COMPARISON/Q>/ 
<VAR/Q>/<#ARGS>/<FN/STRU/Q>/<EDIT/Q>/<FN/REF> 

•l)EilNE/0>:: =jDEFINE ^ 

-j DESCRIBE V <THINGS> [LIKE*] 

(WHAT is/are^j 

»>Does - DO ir DO - Do are also possible. 



-('-;GMPAR1SCN/Q> - ; " ; COMPARE - <THINGS> 




WHAT IS/ARE THE DIFFERENCE( S) BETWEEN <THINGS> 



[HOW] IS/ARE''-''= <THINGS> 



(different from 

israiLAR TO 
/THE SAME 



<THINGS> 



[HOW] IS/ARE»'"'"'' .<THINGS = 



DIFFERENT*** 



** DO or DOES -''DIFFEl^ FROM also works. 

iW::V SIMILAR, THE SAME, DO - DIFFER also work. 



Ri:LEATION/Q>: 



riT, THEY 

IS/ARE / 
^^/^^^ i<FNNAME> \ 

(<STRUCTURES>j 



KINDS of 
A TYPE OF 

etc. 



Trr>T/Q > 



IS 



(<STRUCTURES>) 
- X<FN/TYPE> > 
^DEFINED ) 



r 

\ <FN/TYPE: 
J<STRUCTURES> I 



=>1 



'** The underscore will match anything. In this way, sentences 

like: 'Is H123 on atom?' and "Is F3 defined?' will be recognized, 
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<VAP/0> : 
<-!-N & VRS>: 



WHAT 
) CAN / 

1WILL r 

' ARE 
CAN BE 
MAY BE 
etc. 



-MRC;S >: : = HOW MANY 



Appendix B-2 

CVARIABLE(S)] <FN £ VRS> [THAT ARE <FN£VRS>] 

\ <fnname>\y c 



I I SET 



rBOUNo) 
j SET ( 

ARGUMENTS* 





'FN/STRU/Q 



= WHAT 



<FN/TyPE> 



CALLED BY 



PiT 




HAVE ' 


JFNNAME / 




NEED 


(fntype^ 




_ TAKE _ 



IT, HIM, her) 



IT, HIM, He"!^ 
<FNNAME> 



CALL(S) 

\_ <FNNAME> _i 

(does) Gfnname>) call 

\_ CANj 2^ IT_J 



SOMEHOW 
IN fSOME? WAY 

L C^nyJ 



(^WHO^ 
; WHATJ 



DOES 
. CAN 
) WILL 
( MAY 



CALL(S j<FNNAME> 
IT 



<FNNAME> 
IT, HE, SHE 



CALL 




CALL 



j DOEsJ j<FNNAME>/ 

) CAN V \ IT J 

ris") (<FNNAME>^ 

^AREJ \ IT 



ARCS also works here. 

also possible are CAN, CALL, MAY CALL, etc. 



any! r/^r^i 



used'" 




^ IN 


CALLED 




FROM 






BY 



<FNNAME> ) 
<FN/TYPE> L 
IT, THEM,) 
ITSELF JJ 

<FNNAME> ) 
<FN/TYPE> ( 
IT ( 
THEM J 
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PRINT ^ 

EDIT S <FNnaME> 

PRETTYPRINTj 

/car'^ 

What is the ( CDR i of <STRUCTURES> 

< CADR > 

IT, THEY, THEM 

<CONCEPTS > 
<FN NAME> \ [ AND <THINGS>] 

<FN/TYPE> 
<STRUCTURES > 
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